Natasha 2: Faster Non-Convex Optimization Than SGD

We design a stochastic algorithm to train any smooth neural network to ε-approximate local minima, using O(ε−3.25) backpropagations. The best result was essentially O(ε−4) by SGD. More broadly, it finds ε-approximate local minima of any smooth nonconvex function in rate O(ε−3.25), with only oracle access to stochastic gradients. ∗V1 appeared on arXiv on this date. V2 and V3 polished writing. This paper is built on, but should not be confused with, the offline method Natasha1 [3] which only finds approximate stationary points. When this manuscript first appeared online, the best rate was indeed T = O(ε−4) by SGD. Several follow-up works appeared after this paper but citing us. This includes stochastic cubic regularization [47] which gives T = O(ε−3.5) in Nov 2017, and Neon+SCSG [10, 49] which gives T = O(ε−3.333) in Nov 2017. These rates are worse than T = O(ε−3.25). Our original method also requires oracle access to Hessian-vector products. However, the follow-up paper of Allen-Zhu and Li [10] enables us to replace the use of Hessian-vector products with stochastic gradient computations. We have revised this manuscript in V3 to reflect this change. ar X iv :1 70 8. 08 69 4v 3 [ m at h. O C ] 2 3 Fe b 20 18